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Abstract 

The reduced Markov branching process is a stochastic model for 
the genealogy of an unstructured biological population. Its limit be- 
havior in the critical case is well studied for the Zolotarev-Slack reg- 
ularity parameter a G (0, 1]. We turn to the case of very heavy tailed 
reproduction distribution a = assuming Zubkov's regularity condi- 
tion with parameter /? G (0, oo). Our main result gives a new asymp- 
totic pattern for the reduced branching process conditioned on non- 
extinction during a long time interval. 
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1 Introduction 

A single type branching process describes a population of particles with in- 
dependent and identical reproduction laws. In the Markov branching process 
with continuous time each particle lives an exponential time with mean one 
and at death splits into a random number of daughter particles. If we assume 
that the branching system starts at time zero from a single particle with v 
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daughters, then the whole process is defined by the distribution of the ran- 
dom variable v. In the critical case when the average number of daughters is 
exactly one E[i/] = 1, the limit behavior of the branching process is studied 
under the following Zolotarev-Slack regularity condition (cf. [lO] and [8]). 
The generating function /(s) = E[s'^] is assumed to satisfy 

f{s) =s + {l- sf+^L , < a < 1, (1) 

where L is slowly varying at infinity. This condition is valid with a = 1 if 
the variance of v is finite, while the case < a < 1 is usually referred to as 
the infinite variance case and is well studied in the literature. 

We turn to the less studied case a = 0, focussing on a special class of slowly 
varying functions (which was initially introduced by Zubkov [TTj) 

L{x) ~ (Inx)-^Li(lnx), /3 > 0, x oo, (2) 

where Li is another slowly varying function. Given ([T]) and a = 0, Zubkov's 
regularity condition ^ is equivalent to the next requirement on the tail 
distribution function (see Lemma [1]) 

¥{u >k)r^ (3k-\lnk)-^-^Li{lnk), k ^ oo. (3) 

Notice that in this case E[i/(ln+ z/)^~^] < oo and E[z/(ln+ z/)^+^] = oo for all 
e > 0. This is a consequence of jH Thm 8.1.8]. 

The main characteristic of the branching process is the number of particles 
Z{t) alive at time t. The key issues of the asymptotics of the non-extinction 
probabihty Q(t) = F{Z(t) > 0) as t — oo and the limit behavior of Z(t) 
conditioned on non-extinction in the case a = were recently addressed 
by Nagaev and Wachtel [Ij. They consider the discrete time version of the 
Markov branching process and obtain general limit results without the extra 
assumption ([2]). By repeating Zubkov's arguments pjj p. 607], our results 
could be carried over to discrete time with no change. In Section H] we give 
a direct proof (which is more straightforward than the counterpart of the 
Nagaev- Wachtel proof) of the following result with the extra condition. 

Theorem 1. // ([T]) holds with a = and L satisfies ([2]), then 

Q{t) = exp (-t^/(i+«L,(t)) , (4) 
where Lq{t) is such a slowly varying function as t oo that 

Lj+^(t)~(l + /3)Li(ti/(i+^)L,(t)). 
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Furthermore, there exists a regularly varying function ( see f l2^ ) 



c{t) = (5) 

such that for all x > 

P{Z{t) < e"'=W|Z(t) > 0) ^ 1 - e"""^', t oo. (6) 

There is a striking feature in the asymptotics of Z{t) which was pointed out 
to us by V. Wachtel. Notice that the regular variation index + of the 
scahng function c{t) increases as (3 goes from infinity down to (3 = 1. As the 
reproduction tail becomes heavier this is what we expect to happen, namely 
to have larger asymptotic value for the population size Z{t) at survival. 
What is puzzling about ([6]) however, is that as j3 falls below the threshold 
value 1, the corresponding scaling function c{t) attributes smaller size for 
the surviving population despite the fact that the reproduction tail becomes 
even heavier. 

Some light on this phenomenon is shed by the following seminal results by 
Zubkov [TT]. If r(t) is the time to the most common ancestor for all particles 
alive at time t, then under condition ([T]) with < a < 1 

P(r(t) < tx\Z{t) > 0) ^ X, t-^oo, (7) 

while under the conditions of Theorem [T] 

P(r(t) < tx\Z{t) > 0) aj'^/a+z^)^ t oo. (8) 

The latter means that the ratio T(t)/t is asymptotically distributed over [0, 1] 
with the density function 

0^(x)=/5(l + /3)-Vi/(i+^). (9) 

That is, if (3 is changed towards smaller values, then the time to the most 
recent common ancestor will become shorter and one can expect an eventual 
drop in the size of the surviving population as f3 goes below a certain thresh- 
old value. Why the threshold value should be /3 = 1 is an interesting open 
problem. 

Section [2] presents the so-called reduced branching process describing the 
genealogy of the particles alive at time t. In this section we recall the known 
limit processes for the reduced branching processes in the cases < a < 1 
obtained in ^ and Then we state the main result of this paper. Theorem 



3 



[21 giving a new limit structure as t — > oo for the reduced branching process 
in the case a = and ([21) • Our Theorem [21 is an extension of ([HI) in the 
same manner as the results by [3] and [9] are extensions of ([7]). It is worth 
mentioning that in Theorem [2] we do not loose much generality by assuming 
([2D in the case a = since by Zubkov's proof it actually follows that non- 
degenerate limit distributions of T(t)/t must be either of the form ([71) or ([HI) 
for critical branching processes. In Section [H] we establish some preliminary 
results, and in Section [H we prove Theorems [1] and [2l 



2 Limit theorem for the reduced branching 
process 

Let Z{u,t) stand for the number of particles at time u which stay alive or 
have descendants at a later time t > u. For a given time horizon t the 
process {Z{u,t),0 < u < t} is called the reduced branching process. The 
term reduced reflects the fact that we count only those branches in the full 
genealogical tree that reach to the time of observation t. Clearly, 

F{Z{u,t) = 1) =F{T{t) <t-u), (10) 

where r(t) is the time to the most recent common ancestor. If ([ID holds 
with < a < 1, then the limit theorem for the time to the most recent 
ancestor is extended to the following limit theorem for the reduced branching 
process by [3] and [9J 

{Z{tx,t)\Z{t) > 0) ^ ^.(-log(l - x)), (11) 

where the convergence of the processes over the time interval < a; < 1 holds 
in the Skorohod sense. 

Here the limit is a time transformed Markov branching process Za{t), whose 
generating functions can be written down explicitly. The particles of this 
process live exponential times with mean one and at the moment of death 
produce offspring with the generating function 

fa{s)=S+-{l-s){{l~Sr-l). 

a 

This process is supercritical with the offspring number having mean 
/^(l) = 1 + 1/a and a distribution over = 2, 3, . . . with probabilities 

l{fc=2}, if a = 1, 

F{u^ = k)= { {l + a)T{k-l-a) ^ (12) 
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where T is the Gamma function. It is easily checked that the following 
generating function 

F^{s, t) = 1 - (1 - e-* + e-*(l - 

solves the forward Kolmogorov equation 

dFa{s,t) dFa{s,t) 

thus providing the generating function of Za{t). Note that Zi{t) is the well- 
known Yule process of binary splitting with 



Fiis,t) 



se * 



1 - 1 -e-* s 



implying that the distribution of Zi(t) is shifted geometric for all t. 

To state our main result we introduce a Markov branching process Zo{t) with 
infinite mean for the offspring number. Here the reproduction law is given 

by 

P(^o = fc) = k = 2,3,... (13) 

corresponding to the generating function 

fo{s) = s + {l-s)\og{l-s). 

Thus the part of the formula (1121) given for < a < 1 is also valid for a = 0. 
The generating function of Z^it) is 

Fois,t) = l-il-sr-' (14) 

which can be used to compute 

F{Zo{t) = 1) = e-*. (15) 

Theorem 2. Under the conditions of TheoremU\ the weak convergence of the 
processes over the time interval < s < 1 

{Z{tx, t)\Z{t) >0)^Zo log(l - 

holds in the Skorohod sense as t —>■ oo. 
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From this result it easy to recover ([H]) using ffTUj) and ffT^ . The hmit pro- 
cess Zq (— + /9)~^log(l — x)) gives the following algorithm defining the 
genealogical tree: 

• start with a single particle at time which lives a random time 1 — tq, 
where tq has density function ([9]), 

• at the time 1 — tq split the initial particle into a random number of 
daughter particles according to the distribution ( fT3l) . 

• given To, let each daughter particle, independently of other particles, 
mimic the life of its mother, namely let it live a time {1 — ti)tq, where 
Ti has density ([9]) and then split it using ( fT3|) . 

• given To and ti, let a granddaughter particle live a time (1 — t2)tiTo, 
where T2 has density © and then spht it using ffT^ . and so on. 

This should be compared with a similar algorithm describing the limit process 
in ( JTTl) for < a < 1, where the density ([9]) is replaced by the uniform density 
over [0, 1], and the offspring number distribution ( |T3i) is replaced by ( |T2l) . 

Figure [T] clearly indicates an interesting transformation of the limit law for 
the genealogical tree as the parameter a decreases from 1 to and then at 
a = the new parameter f] goes from 00 down to 0. The a-model has com- 
mon branch length distribution and the value of parameter a determines the 
reproduction law, which smoothly changes from the deterministic splitting 
at a = 1 via (|T2|) to the distribution f|T3l) with infinite mean at a = 0. It 
is easy to verify the stochastic domination property: if < ai < 0:2 < 1, 
then P(z/q,2 > k) < ^{i^ai > k) for all k. This property is nicely illustrated 
by simulations on the left part of Figure [TJ the lower is the value of the 
parameter a the faster is the growth of the genealogical tree. 

At a = when the new parameter f3 takes over the control, the dynamics of 
tree behavior drastically changes. As /3 goes from larger to smaller values it is 
the branch length (and not the reproduction) that undergoes transformation. 
Since the reproduction law (fT3l) is common for all (3 G (0, cxd), we observe 
an opposite development of the tree growth: the closer (3 is to zero (and 
therefore the heavier is the tail of the original reproduction law), the closer 
the splitting times are located to the observation time. Here we have another 
example of the phenomenon mentioned earlier concerning the critical value 
f3 = 1. We observe a growth pattern (this time in terms of genealogical trees) 
which reaches its top and then is followed by a monotone decline. 
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Remarks This development in the tree growth depending on decreasing 
values of (3 indicates that at the region (3 = the linear time scaling for the 
reduced process should be replaced by a non-linear one in agreement with 
[m Thm 4(b)]. 

It was pointed out in [B] that the limit reproduction law for the critical 
reduced branching process with a G (0, 1] is related to the merging law of 
the A-coalescent with A{dx) = (1 — a)x~'^dx. Namely, if Yn is the size of the 
next merger given there currently are n branches, then 



Theorem [2] shows that there is a similar link between the reduced processes 
with a = and the A-coalescent with uniform A, i.e. the Bolthausen- 
Sznitman coalescent (see [5]). 

3 Preliminary results 

If /(s) > s for < s < 1, the function 



is obviously monotone and consequently p{x) = 7r(l — e ^') as well. Let ([I]) 
hold with a = and put g{x) = L{e^), then condition ([2]) is equivalent to 



P(y„ = k) = P(z/„ = k\ua < n). 




g{x) ~ X 



Li{x), X — s> oo. 



(16) 



Note that 




therefore 




(17) 



and it follows from (fT6|) and Thm 1.5.8] that 



0.1 0.2 0.3 0.4 0.5 06 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.6 0.9 1 



Figure 1: Six blocks with ten realizations each of the limit process for different 
parameter values. Three blocks on the left correspond to the values: top 
a — 1, middle a — 0.3, bottom a — 0.1. Three blocks on the right correspond 
to the values: top P — 5, middle P — 1, bottom /3 = 0.2. For /3 = 0.2, some 
trajectories are too close to 1 to be visible. Time to the most recent common 
ancestor is the horizontal distance from the right end of the time interval 
[0, 1) to the point where the trajectory leaves the state 1 = 10°. 
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Lemma 1. Put q{t) = — \nQ{t), then 



p{q{t)) = t, (19) 
q'{t) = g{q{t)). (20) 

Proof. The generating function F{s,t) = E[s^^*^] of the Markov branching 
process satisfies the backward Kolmogorov equation 

^^ = nFis,t))-Fis,t) 
with the boundary condition F{s, 0) = s. It follows that 

7T{F{s,t)) = TT{s)+t. (21) 

Putting s = yields the asserted equality (HM 

n{l-Qit))=t, (22) 

since 7r(0) = and Q{t) = 1 - F{0,t). Furthermore, (EID and ([22]) give an 
important representation 

1- F{s,t) =Q{7r{s) +t). (23) 
After differentiating both sides of (|T9|) we find 



P'ta(.)) = ^ (24) 

which together with ( ITTll implies (|20l) . □ 

Lemma 2. // ([1]) /ioWs wi/i a = 0, then ([2]) implies ([3]). On t/ie oi/ier /iand, 
z/E[i/] = 1, then implies ([T]) with a = and 1^. 

Proof. First notice that 

oo 

i-f{s) = {i-s)j2n^>k)s' 

k=Q 

and similarly, since E[z/] = Yl'k'=o^i^ > k) = 1, 

^T=^=(i-^)f:fEp('^>^))^'- (25) 

A:=0 \i>A: / 
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Therefore, condition ([T]) with a = is equivalent to 



f;f^p(.>»)).- = (i-»)-'L(^) 

k=0 \i>k / ^ ^ 



According to [1, Cor. 1.7.3] the latter is equivalent to 

^P(i^ > ~ L{k), k 



i>k 

or in the integral form 

(•oo 

P(z/ > y)dy ~ L{x), x ^ oo. 
Now given ([2]) we apply [Xi Thm 1.7.2b] to see that 

POO 

J z 

entails 

P(z/ > e')e ~ 2 ^ oo 

which is dH])- To prove the assertion in the opposite direction one should 
apply [H Thm 1.5.11]. □ 

Lemma 3. Under conditions of TheoremUl as x ^ oo 

g[x) ~ , (26) 

X 

9 K^) ~ / X , 27 
q{x) 

P'\q{x)) - f , V (28) 



// we define c(t) by 



if) ^ " iim) • 



then it will satisfy ([5]). 
Proof. In the critical case we have 

i~m = (i-s)EI E 



oo / oo 



irw = I] 1 



k=0 \i=k+2 
oo 



:i-s)^E[z/l|,>,+2}] s\ 



k=0 
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and on the other hand, due to 

k=0 \i>k J 

oo 

= (l-s)2^E[(z/-A;-l)l{,>fc+2}]s^ 



k=0 



These two relations together with L((l — s) ^) = (/(s) — s)/(l — s) yield 



oo 

= - 5^ (E [z/l{,>,+2}] - E [(z/ - A; - l)l{.>fe+2}] ) s' 



k=0 

oo 



fc=0 

thus due to ([3]) as s ^ 1 

L' (j^) ~ -/5(l-s)2^(lnA;)-^-%(lnfc)s^ 

^ '^^ A;=0 

Since due to [H Prop. 1.5.8] 

n 

^(lnA;)-i-%(lnfc) ~ n(lnn)-^-'^Li(lnn), 

A;=0 

we derive from the previous relation applying [H Cor. 1.7.3] 

Y^^'(y^) - -/5|ln(l-.)ri-%(-ln(l-.)). 

Now (1261) follows from the relations g'{x) = e^L'(e^) and Q. 

To derive ( l27j) it is enough to observe that g"(a;) = g'{q{x))q'{x) and use ( |26l) . 
From (EH) we get p"(g(x)) = -q"{x)/q'{xf. This and (EZD give (EH]). The 
last assertion of the lemma is a simple consequence of the basic properties of 
regular varying functions. □ 
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4 Proof of Theorems [T] and 
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The asymptotics (jl]) of Q{t) stated in Theorem [T] follows from (fT^ and (|T^ 
in view of PQ Thm 1.5.12]. 

We prove and Theorem [2] after deriving an expression for the generating 
function of the reduced process 

in terms of F{s, t). The survival probability at time t for a branching process 
starting from a single particle at time u is equal to Q{t — u). Since the total 
number of particles alive at time u is described by u) we can write 

u, t) = F(l - Q(t - m) + Q(t - n)s, m). 

Combining this with the obvious relation 

F{s;u,t) = 1 -Q(t) +E[s^("'*)|Z(t) > 0]Q{t) 

we deduce 

1 - F(l - Q(t - m) + g(t - u)s, u) 



1 



Q{t) 

In view of fl23l) it follows that 



E[.^(-*)|z(t) > 0] = 1 - QW-Qit-i^)i^-^)) + ^) 

= l-exp{g(t) + A(s,t-M))}, (30) 

where 

A{s,t)=p{q{t)-\n{l~s))-t. 

Using (!30l) with u = t we get 

-ln(l -E[s^W|Z(t) >0]) =g(p(-ln(l-s))+t) (31) 

To prove ([6]) we study (I3T1) with s = 1 — e^'^'/^'^*\ where a; is a fixed positive 
number and c{t) is defined by ( |29l) . According to Lemma [3] and (JT8i) 

p(x/c(t))~xi+Vg'(t), t^oo, 
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and therefore, by a Taylor expansion around t, 

-ln(l-E[(l - e-"/^W)^W|Z(t) > 0]) 
= q{p{x/c{t))+t)-q{t) 
= p{x/c{t))q'{t) + p{x/c{t)fO{q"{t)) 

Thus 

E[(l - e-"/"(*))^(*)|Z(t) > 0] ^ 1 - e-"""*^ 

and by the arguments of Darhng [2] and Seneta [7] , or Nagaev and Wachtel 
[1], this imphes that 

P(lnZ(te,t) < xc{t)\Z{t) > 0) ^ 1 - e"^'^", 
which finishes our proof of Theorem [TJ 

The proposed hmit process R{x) = Zo{—j3{l + log(l — x)) in Theorem 
[2] is a time inhomogeneous Markov branching process, and from iHM we find 
that 

E[s^^y^R(x) = 1] = 1 - (1 - s)^^) ,0<x<y<l. 
This yields 



nR{y) = l\R{x) = 1) = ^-^j . (32) 

By similar arguments that led to ( l30l) we have 

E[s^('''*)|Z(M,t) = l,Zt>0] = I - exp{q{t - u) - q{A{s,t - v) + t - u)}, 

for < u < V < t. In order to prove convergence of finite dimensional 
distributions it suffices to show 



g(A(s, xt) + yt) - q{yt) - log(l - s 



< X < y < 1. 



We do this in two steps. First we find the asymptotics of A(s,t) by Taylor 
expansion of the function p around q 

A{s,t) = p(g(t)-ln(l-s))-p(g(t)) 

= -Hl-s)p'{q{t)) + 0{p"{qm 
ln{l -s)/q'{t). 
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And second we do another Taylor expansion, this time of the function q 
around yt 

q{yt + A{s, xt)) - q{yt) = - ln(l - + {q'it))-'Oiq"{t)) 

--Ml -.)(-) . 

according to Lemma [3l This finishes the proof of convergence of finite di- 
mensional distributions in Theorem [21 

To show the convergence in the Skorohod sense, we can now copy the proof 
in |3] almost verbatim. The only difference is that we have fl32p with < 
(3/{l+P) < 1, whereas they in our notation have the expression (1— x) 
for the corresponding probability. The difference in the exponent has no 
consequence for the proof. 

Acknowledgement. We thank V. Wachtel for stimulating discussions of 
his recent paper (1]. 
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